Background

I'm designing the graceful restarting of TiProxy in cloud environment. The expected process is just to:

1. Use a big maxSurge.
2. Use a big terminationGracePeriodSeconds (e.g. 24h).
3. Restart.

However, we cannot modify the terminationGracePeriodSeconds without restarting pods. To be able to restart the existing pods gracefully, we'd better to design another workaround for it.

Design

1. Add a new annotation core.pingcap.com/tiproxy-graceful-shutdown-delete-delay-seconds. With this annotation, the deletion of TiProxy object will have two extra steps before removing the pod: remove the label, wait for several seconds.
2. The annotation will be propagated from TiProxyGroup to TiProxy without rolling restart, so we can modify it for existing resources.

I understand it's not elegant (so I didn't use a spec field to describe it) as it doesn't know when the TiProxy actually has no connection and can exit earlier. IMO, it'd be still better to use a bigger terminationGracePeriodSeconds to gracefully shutdown.

Usage

Patch an existing TiProxyGroup, so it'll wait for a while before deleting the pods:

kubectl --context "$CONTEXT" -n "$NS" patch tiproxygroup pg --type merge -p '{
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "core.pingcap.com/tiproxy-graceful-shutdown-delete-delay-seconds": "20"
        }
      }
    }
  }
}'